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Consulting's Career Development Model; and participants' 
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This study focused on measuring the effectiveness of an Integrative Learning 
workshop provided to 506 human resource professionals in Andersen 
Consulting* s North American practice area. The Performance Change Curve 
was introduced as a framework for self-report data regarding targeted 
• performance change three months after the workshop. Implications are drawn 
for future performance-focused education efforts at Andersen Consulting. 
Keywords : Performance-Focused Learning , Performance Change , Integrative 
Learning, Reflective Learning, Learning Theory, Workplace Learning, Research 
Results 
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Andersen Consulting (AC) has pioneered the development and use of performance- 
focused human resource development programs with its global consulting workforce since 
early 1993 (Nowakowski, 1994; Montgomery, 1996a). In Fiscal Year 1996 (September, 
1995 through August, 1996), Andersen Consulting Education (ACE), Andersen 
Consulting’s internally focused professional development organization, invested more 
than $17,100,000 in development and delivery of performance-focused human resource 
development programs for AC’s global workforce of 41,000 consultants (Montgomery, 
1996b) 1 Measuring the performance return on this investment is recognized as a 
challenge and a growing business imperative. Research data documenting the 
effectiveness of these programs has not kept pace with the demand for and development 
of these programs. In early 1996 AC began collecting feedback survey data from 
participants in these programs using electronic media. This use of technology has made 
gathering research data more cost effective and practical. 

This study focused on the evaluation of the effectiveness of one 8-hour, 
Andersen Consulting performance-focused workshop, the “Applied CDM Learning Day,” 
offered to 506 human resource (HR) professionals in AC’s North American practice area 
between March and September, 1996. 



Background 

In 1995, in order to stay ahead of the growing demand for deeply skilled consulting professionals 
to meet the needs of its global clients, Andersen Consulting (AC) introduced a competency-based 
framework for recruiting, staffing, and developing its global workforce of consulting professionals 
(Montgomery, 1996b). The Career Development Model (CDM) (Montgomery, 1996d) was first 
introduced in AC’s North American practice area in a series of information-focused sessions 
provided to human resource professionals supporting this workforce. It soon became clear that 
these professionals needed additional support to learn to work with this model in their daily 
routines. To meet these needs, a performance-focused workshop was developed in December, 
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1 This only represents a portion of the financial investment made by Andersen Consulting in 
developing its global consultants. AC’s gross income for the Fiscal Year was in excess of $4 
billion and 6.5% was invested in the continuing development of its workforce (Montgomery, 

1996b). 
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1995, using a systems approach to Integrative Learning design (Montgomery, 1996c). The 
conceptual design for this workshop was tested in February, 1996, and the workshop was piloted in 
March with very minor changes. . The design for this workshop was introduced at the 1996 
Conference of the Academy of Human Resource Development (Montgomery, 1996d). 

After completion of the workshop, learners were expected to have moved up the 
Performance Change Curve (Montgomery, 1996a) (see Figure 1) in four areas: 

1) Be more proficient in using the Career Development Model in making HR decisions regarding 
staffing, recruiting, and training. 

2) Work more effectively in supporting other HR functional specialties. 

3) Regularly think about what they are doing on the job and proactively implement ideas for 
continuous improvement in the way they work. 

4) Proactively share learning and improvement ideas with others to help in the development and 
growth of AC’s HR organization. 
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Research Questions: 

’ 1. Can participants use the Performance Change Curve (see Figure 1) as a common framework 
by which to measure targeted performance change? 

2. Will the data regarding growth in targeted performance immediately after the workshop be 
consistent with the results of the “7-days after” survey? 

3. How much growth on the Performance Change Curve did learners experience over the three 
months following the workshop? 

4. Is there a correlation between frequency of personal use of the CDM and learning level on the 
Performance Change Curve in CDM-related targeted performance over the three months 
following the workshop? 

5. What support activities have the learners experienced with regard to working with the CDM in 
the three months following the workshop? 



Methodology: 
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The “Applied CDM Learning Day” was conducted in 21 sessions at cities throughout North 
America during the period March through September, 1996. The number of participants in each 
session ranged from 1 1 to 46. Each of the 506 participants 2 was sent an electronic survey seven 
days after the workshop and three months after the workshop. 221 participants (44%) responded to 
the “7-days after” survey and 101 participants (20%) responded to the “3-months after” survey. 3 
The results of the “7-days after” survey are presented in a separate paper (Montgomery, 1996e). 

Only one version of the “3-months after” survey was used. This survey asked learners to 
place themselves on the Performance Change Curve for each of the four targeted performance 
areas for three time periods: Before the workshop; immediately following the workshop; and three 
months after the workshop. Learners were also asked how frequently they used the CDM and how 
frequently their internal customers used the CDM. The survey then collected frequency data 
regarding use or involvement in specific activities regarding support for their own learning, 
organizational support for learning, and knowledge transfer to internal customers. 

Survey Composition The “3-months after” survey contained 14 questions divided into 
three sections. Questions 1 through 4 captured demographic information. Questions 5 through 9 
captured information and comments regarding application and use of Andersen Consulting’s 
Career Development Model. Questions 10 through 14 captured self-report data regarding growth 
on the Performance Change Curve for each of four targeted performance outcomes. For all 
questions other than typing a response, keywords were identified beside radio-buttons. To select a 
response, the participant would point over the button and “click” the computer’s left “mouse” 
button. The choices were all made by keyword rather than by number. The Lotus Notes mail-in 
database was programmed to convert the selections to numerical responses. Data collected in the 
mail-in database was displayed in “views” matching the questions on the survey. Data displayed in 
these “views” was then exported into a Microsoft Excel spreadsheet where data analysis was 
conducted. 



Results and Analysis 

Targeted Performance 

1. I have become more proficient in using the Career Development Model (CDM) in making 
Human Resource (HR) decisions regarding staffing, recruiting, and training. 

2. I have increased my ability to work effectively in supporting other HR functional specialties. 

3. I regularly think about what I am doing on the job and proactively implement ideas for 
continuous improvement in the way I work. 

4. I am proactively sharing learning and improvement ideas with others to help in the 
development and growth of our HR organization. 

Survey Questions Focused on Targeted Performance Change 

The summary question was: How would you assess your growth in the following areas? 
Possible Responses: 

• Apply/Do it without thinking about it (5) 

• Practice/Do it when I think about it (4) 

• Learn/Do it with help (3) 

• Explore/Don’t know how to do it — yet! (2) 

• Recognize/Here’s something new! (1) 

• Unaware/Blank/Zero Level (0) created by three respondents. 



2 While total of 506 human resource professionals participated in the workshop. 92 participants 
participated in workshops conducted after the cut-off date for the “3-months after” surveys used in 
this study. 

3 For the purposes of the tables in the Results and Analysis section of this paper, the numbers used 
represent both the actual number of respondents and the percentage of respondents. 
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Table 1; Targeted Performance Before the Workshop 



Level on Change Curve 


Performance 1 


Performance 2 


Performance 3 


Performance 4 


Apply = 5 


8 


10 


23 


21 


Practice = 4 


28 


24 


39 


35 


Learn - 3 


21 


24 


15 


23 


Explore - 2 


28 


22 


15 


10 


Recognize = 1 


15 


16 


8 


12 


Unaware - 0 


1 


3 


0 


0 


Number of Responses 


101 


99 


100 


101 


Average Score 


2.83 


2.81 


3.54 


3.43 



Table 2: Targeted Performance Immediately After Workshop 



Level on Change Curve 


Performance 1 


Performance 2 


Performance 3 


Performance 4 


Apply = 5 


13 


17 . 


33 


32 


Practice = 4 


40 


36 


42 * 


34 


Learn = 3 


30 


29 


15 


22 


Explore = 2 


16 


15 


9 


8 


Recognize = 1 


2 


2 


1 


4 


Unaware = 0 


0 


0 


0 


0 


Number of Responses 


101 


99 


100 


100 


Average Score 


3.46 


3.52 


3.97 


3.82 



Table 3: Targeted Performance 3 Months After Workshop 



Level on Change Curve 


Performance 1 


Performance 2 


Performance 3 


Performance 4 


Apply = 5 


33 


29 


43 


39 


Practice = 4 


49 


44 


45 


41 


Learn = 3 


10 


20 


7 


14 


Explore = 2 


6 


4 


5 


5 


Recognize = 1 


2 


2 


0 


1 


Unaware = 0 


0 


0 


0 


0 


Number of Responses 


101 


99 


100 


100 


Average Score 


4.05 


3.95 


4.26 


4.12 



Table 4: Reported Growth in Targeted Performance Immediately After Workshop 



Growth on Change Curve 


Performance 1 


Performance 2 


Performance 3 


Performance 4 


5 levels up 




1 






3 levels up 




. 1 




1 


2 levels up 


10 


7 


6 


4 


1 level up 


48 


50 


31 


29 


0 growth 


38 


39 


63 


65 


1 level down 


5 








2 levels down 




1 






3 levels down 








1 


Number of Responses 


101 


99 


100 


100 


Average Growth 


0.62 


0.71 


0.43 


0.37 



Table 5: Reported Growth in Targeted Performance in the 3 months following Workshop 



Growth on Change Curve 


Performance 1 


Performance 2 


Performance 3 


Performance 4 


3 levels up 








1 


2 levels up 


4 


4 


1 


2 


1 level up 


52 


37 


28 


24 


0 growth 


44 


56 


70 


72 


1 level down 


2 


2 


1 


1 


Number of Responses 


100 


99 


100 


100 


Average Growth 


0.58 


0.43 


0.29 


0.3 



Survey Questions Focused on Use of the Career Development Model (CDM) 

A. How often do you personally work with the CDM in your daily Human Resource (HR) 
activities? (Example: How often do you bring up CDM terms or concepts in your phone calls 
with customers?) 




4 



5 



Monitoring the Effectiveness of a Performance-Focused Human Resource Development 
Workshop at Andersen Consulting Three Months After the Workshop 

B. How often are your Operating Group customers demonstrating understanding of the CDM in 
HR-related conversations? (Example: Partners asking you about staffing by skill track, skill 
domain, and proficiency levels.) 

Possible Responses: Almost always (5); Daily (4); Weekly (3); Monthly (2); Almost never (1); No 
Response. 



Table 6: Use of the Career Development Model 



Question 


Almost Always 


Daily 


Weekly 


Monthly 


Almost Never 


No Response 


Persona] Use 


14 


40 


35 


7 


4 


1 


Customer Use 


0 


15 


36 


19 


25 


6 



Survey Questions focused on Support for Learning to work with the CDM: 

Figure 2 



What are you personally 
doing to increase your own 
ability to work with the CDM 
more easily in your daily HR 
activities? (Example: Working 

with the C-Map Writer (an on- 
line tool) once a week to prepare 
sample skill maps for my 
customers.) 



Personal Support Activities 



48 




m cm —i O 



□ Responses 



Table 7: Personal Support Activities 





Creating my 
own job aids 


Developing job 
aids with 
others 


Using job aids 


Participating in 
CDM-related 
meetings and 
lunches 


Bringing up 
CDM-related 
ideas in 
conversations 


Learning to do 
more with the 
C-Map Writer 
tool 


Other 


Responses 


6 


8 


18 


20 


67 


41 


11 



What is your operating 
group doing to help you learn to 
work with the CDM in daily HR 
activities? (Example: Lunch & 

Learn “Brown-Bags” to share 
CDM-related best practices and 
ideas.) 



Figure 3 



Organizational Support Activities 

59 




jU Responses 



Table 8: Organizational Support Activities 





Job Aids are 
Developed and 
Shared 


Regular Cross- 
Functional Team 
Meetings 


Weekly 

Meetings 


Informal 

Conversations 


Lunch and Learn 
"Brown Bags" 


Other 


Responses 


7 


18 


7 


54 


6 


15 
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What are you and your 
operating group doing to help 
customers learn how to work with 
the CDM in their dealings with 
you? (Example: Taking the time 
to walk a customer through the C- 
Map and proficiency plan when 
speaking about staffing issues.) 



Customer Support Activities 




act. act. act. act. 



□ Responses 



Table 9: Customer Support Activities 





Talking about the 
CDM in routine 
conversations 


Making special 
CDM-focused 
calls 


Asking questions 
using CDM terms 


Making 

recommendations 
using CDM terms 


Other Activities 


Responses 


59 


3 


51 


38 


8 



Conclusions: 

Research Question 1: Can participants use the Performance Change Curve (see Figure 1) as a 
common framework by which to measure targeted performance change? Results of the “3-months 
after” survey indicated that respondents were able to measure their targeted performance change 
against the benchmark offered by the Performance Change Curve, both in terms of growth and 
absence of growth. 

Research Question 2: Will the data regarding growth in targeted performance immediately 
after the workshop be consistent with the results of the “7-days after” survey? 77% of the 
respondents to the “7-days after” survey reported at least “moderate” growth in all four of the 
targeted performance areas. The wording for the four areas was slightly revised for the “3-months 
after” survey. 81% of respondents reported performance on at least the Learn level of the 
Performance Change Curve for all four of the targeted performance areas. 26 % moved to the 
Learn level or above in the CDM-related targeted performance; 24% moved to the Learn level or 
above in the area of working more effectively with cross-functional HR specialties. 13% moved to 
the Learn level or above in the area of reflecting on their work and implementing continuous 
improvement; 9% moved to the Learn level or above in the area of sharing their continuous 
improvement ideas with the organization and with their colleagues. These percentages are 
consistent with “moderate” growth. The data also suggests that the Learn level on the 
Performance Change Curve is a reasonable target for performance change in workshops of short 
duration. 

Research Question 3: How much growth on the Performance Change Curve did learners 
experience in the three months following the workshop? 91% of respondents reported 
performance at the Learn level or above on the Performance Change Curve for all four of the 
targeted performance areas. 9% of respondents reported continued growth to at least the Learn 
level in the CDM-related targeted performance; 11% of respondents reported continued growth to 
at least the Learn level in the area of working more effectively with cross-functional HR 
specialties. 5% of respondents reported continued growth to at least the Learn level in the area of 
reflecting on their work and implementing continuous improvement; 6% of respondents reported 
continuing growth to at least the Learn level in the area of sharing their continuous improvement of 
ideas with the organization and with their colleagues. These results indicate that growth does 
extend beyond the completion date of the workshop. 72% of respondents reported performing on 
the Practice or Apply levels on the Performance Change Curve for all four areas of targeted 
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performance change at the end of the three-month period. (This may reflect a narrowing of focus 
with regard to implementing the targeted performance change.) 

Research Question 4: Is there a correlation between frequency of personal use of the 
CDM and learning level on the Performance Change Curve in CDM-related targeted performance 
over the three months following the workshop? .) Using the Correlation functions available in 
Microsoft Excel, the correlation between these two variables was .0.46 (not statistically significant 
at the P = < .05 level). 

Research Question 5: What support activities have the learners experienced with regard 
to working with the CDM in the three months following the workshop? Possibilities included 
personal activities, organizational activities, and customer-focused activities. 96% of respondents 
reported being involved in one or more activities at the personal level (with 47.5% involved in 
only one activity). 80% of respondents reported being involved in one or more activities at the 
organizational level (with 58.4% involved in only one activity). 58% of respondents reported 
being involved in one or more customer-focused activities regarding the CDM (with 55.4% 
involved in only one activity). 69% of respondents indicated that their customers worked with the 
CDM in interactions with them at least monthly. Data regarding customer-support indicates that 
these activities have a lower priority than personal and organizational activities in understanding 
and working with the CDM. 

The survey data indicates that learners who were actively involved in supporting their 
personal learning and in transferring knowledge to their internal customers also experienced 
growth along the Performance Change Curve for the targeted performance outcomes of the 
workshop. A follow-on survey is planned to ask respondents for the factors they experienced 
which inhibited growth along the Performance Change Curve. 



Implications 

This “3-months after” survey confirmed the results of the “7-days after” survey reported elsewhere 
(Montgomery, 1996e). The Performance Change Curve provided a reference point participants 
were able to use to benchmark their performance change. The electronic survey offered a 
collection approach learners were able to work with. Both the Performance Change Curve and this 
method of data collection and reporting, including both “7-days after” and “3-months after” 
surveys, are already being used to evaluate the effectiveness of other human resource development 
workshops at Andersen Consulting. 



Performance Change 
Curve at the 
Organizational Level 
The Performance 

Change Curve has 
implications at the 
organizational level 
(Montgomery, 1 996a) 
(see Figure 5.) For the 
CDM-related 
performance change to 
become a part of firm 
culture, the organization 
needs to move through 
the learning levels on the 
curve. The need for this 
workshop indicated that 
AC is on the Learn level 



Figure 5 



Performance Change Curve 

Experienced at the Organizational Level 



6 


Integrate 








- 


Rejection = Choice to 
change to something 
else. 


5 


Apply 




Commitment / 




4 


Practice 




fo rhanpp / Action 







- 


Rejection = Choice to 
return to previous 
comfort zone. 


3 


Learn 




/ 


2 


Explore 




/ Acceptance 


/ 


- 


Rejection = Choice to 
remain in current 
comfort zone. 


1 


Recognize , 






B Awareness 



Time and Reflection 
Experienced at the organizational level: 

Vision: Change required to I . I Implement in selected groups. 

I I b 



move beyond existing limits! 

Explore the proposed changi 



J Build organizational acceptance. 



Note: Comfort zone may not 
be stable. Habits are part of 
the comfort zone. 



2 

"T - Endorse the change and 
provide logistics support. 



_ I Implement as an organizational 
J | standard, reinforced in appraisals. 

6 j it's a part of organizational culture! 



7 



BEST COPY AVAILABLE 



Monitoring the Effectiveness of a Performance-Focused Human Resource Development 
Workshop at Andersen Consulting Three Months After the Workshop 

on the Performance Change Curve. More logistics support for implementing the CDM is required. 
The data regarding use by customers and the response to Research Question 5 indicate that support 
for this change is not uniform across the offices the participants were drawn from. 

Organizations can work with the Performance Change Curve to evaluate their level of 
support and commitment to targeted performance change. Further research using the curve is 
needed to determine the usefulness of its details in tracking performance change at an 
organizational level. 

The Performance Change Curve may also serve as a common benchmark for sponsors of 
human resource development programs to use to target performance change coming from human 
resource development programs. Participants in these programs and supervisors can use the same 
benchmark to identify the effectiveness of the programs and of the logistical measures in place to 
support performance change in the workplace. 

How this Research Contributes to New Knowledge in HRD: 

This research documents the effectiveness of a performance-focused human resources development 
workshop developed by and for Andersen Consulting’s human resource professionals. This builds 
on the work introduced at the 1996 Conference of AHRD and documents the effectiveness of 
electronic surveys as a medium for data collection. The Performance Change Curve was used as a 
benchmark for self-report feedback by respondents to the survey and was adapted for use at the 
organizational level as well. These two versions of the Performance Change Curve represent 
innovations in thought in HRD literature. This study also builds on an earlier study (Montgomery, 
1996e) to document the effectiveness of an Integrative Learning systems approach to designing 
human resource development programs. 
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